IN THE UNITED STATES PATENT AND TRADEMARK OFFTCF 



CERTIFICATE OF EXPRESS MAIT.rNn 
I hereby certify that this paper and the documents and/or fees referred to as 
attached therein are heing deposited with the United States Postal Service 
on November 28, 2000 m an envelope as "Express Mail Post Office to 
Addressee" service under 37 CFR §1.10, Mailing Label Number 
EL631004128US, addressed to the Assistant Commissioner for Patents, 
Washington, DC 20231. 



Washington, DC 
Kristina Gomez 



Attorney Docket No. : DIGEPO 1 3 
First Named Inventor: CROSBY 



UTILITY PATENT APPLICATION TRANSMITTAL (37 CFR. § 1.53(b)) 

(Regular application claiming priority of a provisional) 



Assistant Commissioner for Patents 
Box Patent Application 
Washington, DC 20231 



Sir. This is a request for filing a patent application under 37 CFR. § 1 .53(b) in the name of inventors: 
MATT CROSBY & DAVID WILKINS 

For TECHNIQUES FOR GENERATING A DISTRIBUTED LOW-RESOLIJTTON DIGITAL 
IMAGE CAPABLE OF VIEWING IN ANY RESOLUTION 



Priority of U.S. Provisional Application No. 60/172,301 filed on December 16, 1999 is claimed 
under 35 U.S.C. § 119(e). 

Apphcation Elements: 

r" ^1 49 Pages of Specification, Claims and Abstract 
I^K] 9 Sheets of formal Drawings 
I I Combined Declaration and Power of Attorney 
I I Separate Declaration 

Accompanying Application Parts : 



I Do not publish this application. Nonpublication Request is attached. 

Assignment and Assignment Recordation Cover Sheet (recording fee of $40.00 enclosed) 
Separate Power of Attorney by Assignee 
37 CFR 3.73(b) Statement by Assignee 
Information Disclosure Statement with Form PTO-1449 
I I Copies of IDS Citations 
j I Preliminary Amendment 
[X^l Return Receipt Postcard 



(Revised 1 1/00, Pat App Trans 53(b) Prov Priori) 



Page 1 of 2 



PLEASE DO NOT CHARGE A FILING FEE AT THIS TIME 



General AuthorizatioTi for Petition for Extension of Time r37 CFR SI .136;> 

Applicants hereby make and generally authorize any Petitions for Extensions of Time as may be needed 
for any subsequent filings. The Commissioner is also authorized to charge any extension fees imder 37 CFR 
§1.17 as maybe needed to Deposit Account No. 500388 (Order No. DIGIP013). 



Please send correspondence to the following address: 



Customer Number 022434 j 

BEYER WEAVER & THOMAS, LLP \ a--* a 

P.O. Box 778 i 22434 

Berkeley, CA 94704-0778 [ patent and trademark of 

Telephone (650) 961-8300 ^ 

Fax (650) 961-8301 ' 



Michacj^ Ferrazano 

Registration No. 44,105 



(Revised 1 1/00, Pat App Trans 53(b) Prov Priort) 



Page 2 of 2 



Techniques for generating a distributed low-resolution digital 

EVIAGE capable OF VIEWING IN ANY RESOLUTION 
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Cross Reference to Related Applications 
This application claims benefit of priority under 35 U.S.C. 1 19(e) of U.S. 
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1999 and entitled "TECHNIQUES FOR USING DATA ASSOCIATED WITH A DIGITAL IMAGE 
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BACKGROUND OF THE INVENTION 
1 . Field of Invention 

The invention relates generally to digital image processing systems. More 
particularly, methods and apparatus for selectively processing a digital image are 
disclosed. More particularly, the invention provides techniques that allow a low- 
resolution image to be generated in a standard format (such as JPEG, TIFF, or PNG) so 
that anyone can view the resultant image, but with the knowledge of where its distributed 
pieces (i.e. digital negative and edit hst) reside. 
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2. Description of Relevant Art 

When a user performs image operations using digital image processing programs 
such as Adobe Photoshop™ or PhotoDeluxe™, image operations are performed directly 
on the raw pixels of the image. Since most imaging applications only perform image 
operations on one resolution, usually the highest resolution, these operations are 
sometimes very slow, even on the fastest computers. If an application could work at a 
lower resolution for display purposes, the processing time would significantly decrease, 
thus increasing the productivity of the user. While it is sometimes possible for an 
application to work on lower-resolution image data, when the image with all the applied 
image operations is to be saved, the full-resolution image data must be processed at that 
time. If this step were not performed, the saved image would only contain low-resolution 
image data. While this is one option, this is not desired since it would not be possible to 
obtain an image rasterized at a higher resolution. 

In general, an application is forced to generate one resolution (usually the 
highest), even though lower-resolution data could be used since the output may be 
targeted to a low-resolution display device and not a high-resolution printer. 
Unfortunately, it is not possible for the apphcation to know apriori how the image will be 
used in the future, thus a high-resolution image is usually created. 

A more desirable solution would be if a low-resolution (i.e. resultant) image 
could be generated based on the sequence of image operations, but with "knowledge" 
about the original high-resolution image (i.e. the digital negative) as well as the list of 
image operations (i.e. the edit list). The digital negative is the user's original, unmodified 
image stored at the highest resolution possible. The edit list is a sequence of image 
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operations or image transforms that are to be applied to an image. The resultant image is 
the result of applying the specified edit list applied to the digital negative at some 
specified resolution, which includes a reference (i.e. internal or external link) to the 
digital negative along with the optional edit list. 

There have been recent developments in technology that have attempted to solve 
this problem. An imaging appUcation that supports FlashPixTM technology, developed by 
a consortium of companies comprising of the Digital Imaging Group (DIG), can display 
and manipulate on-screen images at a lower-resolution, but retains access to the higher- 
resolution image data. The FlashPix™ file contains the following items: (a) the original 
higher-resolution image data, (b) a very limited set of image transforms (such as. rotation, 
translation, cropping, color twist, blur/sharpen adjustment, brightness/contrast 
adjustment), and (c) an optional resultant image. While this appears to resolve the 
problems hsted above, there are several limitations. The most important of which is that 
the original image data, the image transforms, and the optional resultant image are stored 
together in one central file. This does not provide for a distributed imaging architecture 
as describe by the invention, nor is it a lightweight solution needed for transmission of 
images in a low-bandwidth network environment. Additional limitations of FlashPix are 
that the file format is based on a proprietary, complex format referred to as COM / 
Structured Storage which, in the current form used by FlashPix^M, does not support the 
ideas put forth by this invention. Further, the image transforms supported are limited to a 
few operations. The solution described by this invention provides a much more flexible 
model. 
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other attempts have been made that try to remedy the problem outhned above, but 
with limited success. The Internet Imaging Protocol (IIP) has been developed that 
provides for optimized access and display of a FlashPix files over the network. When 
using IIP, a FlashPix file that resides on a serve can be efficiently rendered. A client (i.e. 
Web browser) must contain executable code (such as a Java applet, ActiveX control, or 
browser plug-in) that allows it to request portions of the FlashPix file. This involves 
downloading of fhlly rendered tiles or portions of the image at different resolutions. 
While this does provide for efficient display of a FlashPix file over a low bandwidth 
network connection, it does not solve the generate problem of linking the low-resolution 
resultant image file back to the high-resolution digital negative and edit list. IIP is 
simply a method for serving up and viewing large FlashPix files over the network in an 
efficient manner. 

Other variants of FlashPix exist, including LivePicture's FITS technology. The 
FITS format allows the image operations to be performed on low-resolution image data 
and saved, thus eliminating the need to do a time consuming render of the high-resolution 
image data when saving the file. The re-rendering at high resolution is done once the 
output resolution is determined. However, it does not solve the general problem of 
Unking the low-resolution resultant image file back to the high-resolution digital negative 
and edit list. 

What is desired is a method or system that allows a low-resolution image to be 
generated in a common format (such as JPEG, TIFF, or PNG) so that anyone can view 
the resultant image, but with the knowledge of where its distributed pieces (i.e. digital 
negative and edit fist) reside. 
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SUMMARY OF THE INVENTION 

The invention relates to an improved method, apparatus and system that allows a 
low-resolution image to be generated in a common format (such as JPEG, TIFF, or PNG) 
so that anyone can view the resultant image, but with the knowledge of where its 
distributed pieces (i.e. digital negative and edit Ust) reside. 

In one embodiment, in a peer-to-peer connected environment, a method of sharing 
a multi-media object between a first node and a second node is disclosed. At the first 
node, a digital negative of the multi-media object is stored, wherein the digital negative 
represents the multi-media object at a first resolution. The digital negative is modified to 
form a resultant image at a second resolution, an edit Hst based on the modifying is 
associated with the resuUant image, the edit list is linked to the digital negative. At the 
second node, the resultant image is fetched at the second resolution, an output resolution 
't of the resultant image is determined, the resultant image is converted to the determined 

output resolution and format, and the resultant image is output at the determined output 
resolution. 

These and other advantages of the present invention will become apparent upon 
reading the following detailed descriptions and studjdng the various figures of the 
drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The invention, together with further advantages thereof, may best be understood 
by reference to the following description taken in conjunction with the accompanying 
drawings in which: 

5 Fig. 1 shows a block diagram of a digital image processor system in accordance 

with an embodiment of the invention is shown. 

Fig. 2 illustrates a particular implementation of the image processing engine 
shown in Fig. 1. 

Figs. 3A and 3B are examples of resultant proxy image files and associated edit 
10 list files in accordance with an embodiment of the invention. 

Fig. 4 shows a distributed system in accordance with an embodiment of the 
invention. 

Fig. 5 shows a digital camera system in accordance with an embodiment of the 
invention. 

15 Fig. 6 shows a flowchart is shown detaihng a process carried out by the digital 

image processing engine shown in Fig. 1. 

Fig. 7 illustrates a flowchart detailing a process whereby a user creates a 
intelhgent "e-card" in accordance with an embodiment of the invention. 

Fig. 8 illustrates a process for facilitating efficient video editing in a distributed 
20 environment in accordance with an embodiment of the invention. 
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Fig. 9 illustrates a computer system or any imaging- enabled computing appliance 
/device that employed to implement the invention. 
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DETAILED DESCRIPTION OF THE EMBODIMENTS 
Broadly speaking, the invention relates a novel approach and mechanism that 
allows a low-resolution digital image to contain a link to an associated original digital 
negative and edit hst. Unlike conventional approaches to image editing, the distributed 
nature of the invention provides substantially improved flexibility. More specifically, the 
entire high-resolution image must not always be rasterized or distributed, since through 
this invention, it is possible to easily obtain access to the digital negative and edit-Hst of 
operations, thus allow additional resolutions to be generated on-demand as needed by an 
application or output device. Furthermore, in a distributed system utihzing, for example, 
the Web, CD media, or directly from a digital camera or other imaging appliance device, 
the invention provides that a user's digital negative can reside anywhere within this 
distributed system. In this case, a low-resolution image object can be distributed that will 
facilitate very fast download and display since it may be of a much smaller size compared 
to the original high-resolution image. 

The image object may contain a proxy raster image, or resultant image, of its 
contents in any industry standard format (i.e. JPEG, TIFF, PNG, etc) provided on- 
demand rasterization is not available. The proxy raster image can be a fully rendered 
image of the digital negative with the image operations specified by the edit list applied 
at some resolution. This allows for very fast download and immediately display of the 
image, since it most likely will be of a lower resolution with possibly a higher 
compression ratio. In the described embodiment, a low-resolution (i.e. resultant) unage 
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could be generated based on the sequence of image operations, but with "knowledge" 
about the original high-resolution image (i.e. the digital negative) as well as the list of 
image operations (i.e. the edit list). The digital negative is the user's original, unmodified 
image stored at the highest resolution possible. The edit Ust is a sequence of image 
5 operations or image transforms that are to be appUed to an image. The resultant image is 
the result of applying the specified edit hst apphed to the digital negative at some 
specified resolution, which includes a reference (i.e. internal or external link) to the 
digital negative along with the optional edit list. 

Further, in a preferred embodiment, it will be in an industry standard format, thus 
10 allowing it to be displayed by most apphcations and Web browsers. More importantly, if 
the digital negative or edit list associated with the proxy image are not available, the user 
will still have a preview of what the image composition should look like. This allows a 
user to view the proxy image (or image composition) at a given resolution, using a 
standard apphcation, but retains its ability to be rasterized at a higher resolution at a later 
15 time, based on specific application or device requirements. It is important to note that 
this invention works within the framework of existing industry standards, thus allowing 
any application to view the proxy image if the application does not have support for the 
architecture outlined by this invention. 

It should be noted, when an application displays the resuhant proxy image, either 
20 because the original image cannot be re-resterized or the apphcation requires immediate 
display of the resultant proxy image, the proxy image might not actually match the 
resolution of the output device. If this occurs, the resultant proxy image can easily be 
resampled to match the resolution of the output device, a process well known in the art. 
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This is particularly advantageous when viewing images over a network. If two 
end-user computers are configured in a peer-to-peer connected environment, such that 
sharing of photos and other multi-media objects are permitted, this invention will create a 
much more desirable experience, since only low-resolution photos need to be transmitted. 
5 It is only when a user needs access to higher resolution image data, such as when printing 
or zooming into an image, does the higher resolution data need to be optionally accessed. 
Even in a situation where a user is viewing photos from a central photos sharing service 
on a network, such as Zing.com or Kodak's PhotoNet, this invention will permit users 
quick access to the lower-resolution image data, but retains the ability to access higher 
10 resolution image data. 

It should be noted that an additional advantage of the invention is that the 
resultant proxy image can be very small and dependent upon a specific device. For 
example, the resultant proxy image derived from a set of edit operations apphed by a 
digital camera having a small LCD display may in fact be a low-resolution thumbnail 
15 image. In this way, the resultant proxy image will be properly displayed and optimized 
for the LCD display on the camera. By using a linked edit Ust, the editing operations can 
be applied on the fiill resolution version of the original high- resolution image (i.e., the 
digital negative) at a later time whenever a higher-resolution resultant image is needed. 
Clearly then, two of the benefits from this approach are that the image processing 
20 operations are very quick since only those pixels identified by the editing operations at a 
given resolution are processed resulting in a smaller resultant proxy image thereby 
preserving in-camera memory storage space. 
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Based on the editing operation, the software application may require a much 
smaller number of pixels from the digital negative if the editing operation can be 
performed in a resolution-independent manner. This provides for more efficient CPU 
utilization for processing of operations. Additionally, when the actual image data of the 
digital negative is requested in a distributed environment, higher network utilization is 
possible since lower-resolution image data can be transmitted between two computing 
devices provided the output device is lower resolution than the original digital negative. 

However, for cases in which the editing operation is not resolution-independent, 
the software performing the operation may choose to access more pixels from the digital 
negative for the editing operation, but resample (reduce the number of pixel in) the output 
image after the editing operations have been performed, thus resulting in smaller resultant 
proxy image. 

In general, the "edit Ust" is normally associated with a resultant image, such that 
the resultant image can be recreated at a desired resolution as needed. This "edit list" 
contains all the necessary information about how to perform this reconstruction. It will 
include the reference(s) to the digital negative(s) and any additional multimedia assets 
needed for this reconstruction. It should be noted that several multimedia assets 
(including several digital negatives such as still images, video images, vector artwork, 
test, etc.) might be required for this reconstruction. This is because the resultant image 
may be a composite of more than one multimedia asset. 

It will also contain an optional Hst of editing operations that must be performed. 
The "edit Hst" can be considered both the "glue" that identifies all digital negatives and 
other multimedia assets that are needed to reconstruct the resuhant image and the "script" 
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on how the digital negatives and other multimedia assets are rendered such that the 
resultant image can be recreated, at the current resolution of the resultant image or at a 
different resolution. 

It should be reahzed that an "edit hst "can take the form of a variety of different 

embodiments, three of which are described below. This is meant to describe different 

variants of an edit list, but not an all-inclusive list. The first embodiment is a hst that 

describes a set of individual operations that are to be performed on a given photo. These 

edit operations can be applied to a photo to generate the resuhant image at the desired 

output resolution. The advantage of this stmcture is that new edits can be added to the 

end of the list or removed from the end of the list through an "undo" operation. A first 

sample piece of XML code is included for illustrative purposes: 

<!-- Edit hst example that consists of a list of operations applied ~> 
<!-- This example demonstrates when the edit list is a list of sequential 
operations ~> 

<!- that is applied to the digital negative. In the example below, an 
operation can — > 

<!- be apphed more than once, and in any order. This follows the 
conventional ~> 

<! — ^model of a list of operations as entered by the user in which the hst 
can be ~> 

<! — at a later time undone if desired (via an undo operation). -> 
<PICTURE-IQ-EDIT-LIST> 

<DIGITAL-NEGATIVE-LINK type="highest_resolution" height="1600" 
width="1200" 

href="www.pictureiq.com/samples/nicholas.jpg" content- 
type="image/jpeg" /> 

<EDIT-SCRIPT type="sequential_list_of_operations"> 

<!-- auto fix is an operation that is appUed to the digital negative, and 

does ~> 

<!-- not take any parameters ~> 
<EDIT-COMMAND type="auto_fix7> 

<!- color_adjustment is apphed to the digital negative, with the following 
parameters — > 

<EDIT-COMMAND type="color_adjustment" red="10" green="-20" 
blue="50" /> 
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<!-- next a warp is applied witii the following parameter ~> 

<EDIT-COMMAND type="warp" amount="20" /> 

<!- finally, another color adjustment command is specified ~> 

<EDIT-COMMAND type="color_adjustment" red="-10" green="00" 

blue="50" /> 

</EDIT-SCRIPT> 



An alternative embodiment is to use this hst to represent a state description of 
each operation applied to the photo, in which certain operations can be turned on or off. 
Each operation can still have an optional parameter list associated with the command, but 
there is a single "state" for the operation. It is either "on" or "off, and if "on", it can 
have a parameter associated with it. The advantage of this structure is that the operations 
and the interaction between the operations are clearly defined by the structure, thus 
providing consistent resuhs regardless of the order that the user has applied the 
operations. Either the XML syntax could define the order in which the operations are 
performed, similar to the first embodiment, or the application can define it, independent 
of the order in which they appear in the XML file. The key point is that each operation is 
either "on" or "off and would normally not appear more than once on the edit hst. A 
second sample piece of XML code is included for illustrative purposes: 



<!-- Edit hst example that consists of a state description of available ~> 
<!— operations each operation is either "on" or "off, with its specified ~> 
<!- parameter list each operation should only be specified once and the -- 

> 

<!-- order in which it is applied can be strictly specified by the apphcation 
— > 

<PICTURE-IQ-EDIT-LIST> 

<DIGITAL-NEGATIVE-LINK type="highest_resolution " height="1600" 
width="1200" 

href="www .pictureiq.com/samples/nicholas.jpg" content- 
type="image/jpeg" /> 

<EDIT-SCRIPT type="state_description_of_operations"> 
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<!_. warp is specified, with its parameter, but is currently not enabled ~> 
<!- (active). If the user enabled warp again, the appUcation could --> 
<!-- use either a preset value or the one specified in this file (20), ~> 
<!-- as required by the appUcation. ~> 

<EDIT-COMMAND state="off ' type="warp" amount="20" /> 

<!- color adjustment and auto_fix are both enabled and applied to the ~> 

<!-- digital negative. ~> 

<EDIT-COMMAND state="on" type="color_adjustnient" red="10" 
green="-20" blue="50" /> 

<EDIT-COMMAND state="on" type="auto_fix"/> 

</EDIT-SCRIPT> 

</PICTURE-IQ-EDIT-LIST> 



A third embodiment combines the benefits of the first two embodiments. The edit 
list not only represents the state description of operations, but also includes the ability for 
one or more of the operations that are part of the state description to be a fist of individual 
operations that are to be apphed to the photo, thus merging the two methods 

It should be noted, this architecture also allows an apphcation to describe access 
to multiple resolutions of a digital negative from a single file. For example, multiple 
resolutions of the same image (at different resolution) could be externally linked via the 
"edit Ust" that describes the cross-reference between the various distributed image files. 
In this case, the "edit list" is the "glue" that holds the structure indicative of how each file 
is interrelated, but not necessarily the "script" that describes additional editing operations, 
here the "script" portion of the edit list is empty or null. For example, in one 
implementation, the edit list includes a pointer that points to a remote image file having a 
higher (or lower) resolution than the digital negative or, in some cases, a resuhant proxy 
image. 

The capabihty for linking an edit hst(s) from a low-resolution proxy resultant 
image provides substantial advantages for e-comtnerce applications. For example, photo 
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content providers can distribute lower-resolution images for free via the Web or a CD 
that contain references back to the original high-resolution digital negative (s) that can 
reside either on a Web site, CD media, or other removable media. Photo Web sharing 
sites can also use this invention to enable users to link photos, or photo greeting cards 
5 (and other compositions such as multi-page albums) to the high-resolution image data 
and an optional "script" within the edit hst for rendering of the photo (or card or album) 
at a higher-resolution. 

An added security feature could also be provided that allows the high-resolution 
image to be locked with, for example, an encryption key to prevent unauthorized access 
10 unless the customer has actually purchased the rights to the given photo. It is a benefit of 
this invention that the "key" for how the high-resolution digital photo can be accessed is 
specified by the edit list (either embedded or linked) of the low-resolution proxy image. 
In addition, it has a further advantage that this meta-data could also include ordering 
information such as price, restrictions and the like. 
15 It should be noted that an "edit list" of a multimedia asset, such as a digital image, 

can take many forms, such as for example, a single file representative of one digital 
negative and one separate XML file associated with the edit Ust for a given resultant 
image. In other cases, the edit hst can be part of a file, referred to as a catalog file, that 
includes several different "edit hsts" each being associated with a different multimedia 
20 asset. In one implementation, a catalog file can include several digital negatives (and 
their corresponding resultant unages) and a single XML file that contains several 
different XML edit hsts, one for each resultant image. In this way, the catalog file is an 
efficient approach for maintaining edit hsts for a number of images in a single file. 
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The proxy image may also be obscured or "stamped" to render it unsuitable for 
printing or distribution, while still retaining the capability to access the original image if 
the user purchases such rights. The proxy image could also contain either a visible or 
invisible watermark, thus providing an additional form of identification and protection to 
5 the image. 

It should be noted that both the digital negative and the resultant proxy image can 
be any industry-standard image formats (i.e. JPEG, PNG, GIF, etc) or even a proprietary 
file format (although not in the preferred embodiment) since the edit list and links 
between the images can be encoded in private meta-data embedded inside or externally 
10 linked to the image files. In this way, any application regardless whether or not it is 
cognizant of the invention is able to open and view any image that contain the private 
meta-data, since the resuhant image contains a rasterized version of the image, albeit 
possibly a lower rendered resolution. Of course, however, applications that do recognize 
and support the invention will have the ability to re-render the image, as necessary, to 
15 achieve any desired result. 

The invention provides for the ability to efficiently download lower-resolution 
images from the Web, but preserves the ability to be rasterized at a higher resolution 
either locally or remotely at a later time when needed, thus preserving the bandwidth. 
In one embodiment, it allows a user to quickly download a low-resolution 
20 resultant image in an industry standard format (such as JPEG, PNG, TIFF, or others), but 
maintains the knowledge about how a higher resolution image can be created. In this 
case, the low-resolution resultant image has associated with it an edit-list of operations 
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that must be applied to the digital negative. This allows for quick viewing, but with the 
option to obtain higher-resolution image data as needed. 

It should be noted that this invention can be used to enhance the transmission of 
image data, no matter what the environment or transport medium. For example, 
5 downloading of user' s images from a cenfral Web site (such as Zing.com, PhotoNet, and 
other on-line photo communities / providers). This invention can also facilitate much 
improved transfer of images over a network in which two end user's computers are 
connected in a direct peer-to-peer fashion. Further, this invention can be used equally 
well, when transmitting images over a wireless network coimection. 
10 In another embodiment, a user can download a low-resolution "Intelligent e- 

Card" from a Web site represented as an industry standard JPEG that contains the 
rasterized composition of photos, cards, text, and any other multimedia assets. While the 
JPEG proxy image may be adequate for on-screen viewing or a preview print, the proxy 
image is embedded with knowledge about how the high-resolution artwork and/or photos 
15 can be accessed and composited, providing the apphcation with the necessary 

information for the rasterization at a higher-resolution. The benefit is that a very small 
sized proxy image (such as an industry JPEG, PNG, TIFF, or other format) can be 
initially downloaded for review very quickly and at a later time, higher-resolution data 
can be accessed. 

20 Assuming that a higher-resolution "e-card" is to be created, this invention permits 

the creation of the high-resolution data either locally on the cUent computing device or on 
a remote computing device (such as a server computer). In the case that rendering occurs 
on the client computing device, higher-resolution image data is obtained from the digital 



DIGIP013/MJF 



17 



PATENT 



negative (s) and the edit lists are processed generating a higher resolution "e-card". 
Alternatively, the client computing device could request that the rendering be performed 
remotely (such as on a high-performance server), but where the rendered higher 
resolution "e-card" be subsequently transferred to the chent computer in its final form.It 

5 should be noted that this architecture could be configured to meet the requirements of 
many different e-commerce sites. The low-resolution "Intelligent e-Card" could be 
watermarked, thus encouraging the user to purchase a higher-resolution representation. 
Further, since the low -resolution resultant image has the knowledge about how to 
recreate a higher-resolution version, the e-commerce site could require, when this high- 

10 resolution data is accessed, that the user purchased the rights to access and use the high- 
resolution data. 

It should also be noted, this is not limited to greeting cards, but could also be 
applied to the creation and distribution of albums, calendars, and virtually any other 
creation that involves incorporation of photographs or other image data into a 

15 composition. 

It should also be noted that in all these embodiment, several performance 
optimizations can easily be attained. In particular, when a photo or composition (such as 
a card, calendar, or album) is printed through a remote location, the customer (end user) 
may not even need to transmit much data from their local computing device. Although 

20 the customer may only have direct (local) access to a low-resolution proxy image, the 
resolution that best matches the output device (such as a high-resolution printer) will be 
rasterized based on the original digital negative and the edit list stored with the proxy 
image. More importantly, if the high-resolution image resides on the Web (such as 
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Zing.com or PhotoNet) and output is going to a remote device (such as a printer at 
Kinkos), very little client bandwidth is required from the local user's computing device. 
Since the resultant image has associated with it the edit hst and references to the digital 
negative(s) data, only that small amount of information must be transferred from the 
5 user's local computing device to the remote device (such as a print service). 

In a particular video-based embodiment described below, the user is able to 
download and edit a low-resolution digital video stream from the Web. The video editing 
commands can be passed to the server for processing on the ftiU-resolution video stream. 
The commands are associated with the low-resolution digital video stream that maintains 
10 a reference back to the high-resolution video sfream on a server. At a later point in time, 
the editing commands can be processed on the high-resolution video steam remotely. 
This requires only a minimal amount of data to be transmitted (only the editing 
commands), resulting in less bandwidth and processor requirements on the cHent side. 
In this way, the invention provides a very powerful distributed multi-resolution, 
15 and non-destructive imaging model. It is also platform, operating system, and device 
independent. Furthermore, it is extremely lightweight and compatible with many other 
standards. There are many unique aspects to this invention that will be realized after 
fiirther review. The distributed characteristic of this invention is the unique aspect that 
provides the real flexibility and power not found in any application or system today. 
20 While this invention was originally developed for a digital imaging (still and video) 

consumer electronics devices, its applicability spans any computing device that performs 
operations on multimedia content. 
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Referring now to Fig. 1, a block diagram of a digital image processor system 200 
in accordance with an embodiment of the invention is shown. Digital image processor 
system 200 includes an image source 202 arranged to provide any number of digital or 
analog image input signals for processing. The image source 202 can provide a digital 
5 image stream that can take the form of a still image (having a format such as JPEG or 
TIFF) as well as video from, for example, a digital television (DTV), a DVD, a CD, set 
top box (with satellite DSS or cable signal ), the Internet (also commonly referred to as 
the WEB), and the like. In this way, a digital image source 204 can provide any number 
and type of well-known formats, such as, JPEG, BMP, TIFF, BNC composite, serial 
10 digital, parallel digital, RGB, or consumer digital video. When the digital image source 
204 is a digital television, the input signal is a digital video signal having any number and 
type of well known digital formats such as, SMPTE 274M-1995 (1920 x 1080 resolution, 
progressive or interlaced scan), SMPTE 296M-1997 (1280 x 720 resolution, progressive 
scan), as well as standard 480 progressive scan video. Alternatively, as an analog image 
15 source 206, the analog signal provided includes analog signals derived from, for example, 
an analog television, still camera, analog VCR, DVD player, camcorder, laser disk 
player, TV tuner, scanner, set top box (with satellite DSS or cable signal), game 
machines, imaging kiosks, as well as the WEB, and the like. 

In order to convert the analog signal provided by the analog image source 206 to 
20 an appropriate digital signal, an analog-to-digital converter (A/D) 208 coupled to the 
analog image sowce 206 converts an analog voltage or current signal into a discrete 
series of digitally encoded numbers (signal). In this way, a digital image data word 
suitable for digital processing is thereby generated. Any of a wide variety of A/D 
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converters can be used, such as for example those manufactured by: Philips, Texas 
Instrument, Analog Devices, Brooktree, and others. 

In the described embodiment, a selector unit 209 selects the digital image stream 
from either the digital image source 204 or the A/D converter 208 to form an input digital 
5 image stream 210 to a digital image processing engine 212. Such an engine can be 

implemented as described in more detail with reference to Figs. 4-6. In some cases, the 
input stream 210 has embedded therein digital image in the form of a digital negative also 
referred to as a reference. In other cases, however, the digital image embedded in the 
input stream 210 can take the form of aresultant proxy image 300 (i.e., modified image 
10 file) having either an embedded edit hst file 302 as illustrated in Fig 3 A or having a 

pointer that identifies an external edit list file 304 as shown in Fig. 3B. In either case, the 
edit list files 302 and 304 provide a pixel by pixel image processing instruction set 
operable on corresponding pixels in the associated digital negative. 

In the described embodiment, an external edit list source 213 provides external 
15 edit list data to the digital image processing engine 212. The external edit list source 213 
can represent any number and type of devices suitably arranged to provide appropriately 
configured edit Hst data used to modify an associated digital image. Such devices 
include, but are not limited to, user interface (UI) controllers (at the control of an 
apphcation or other device), a resuUant proxy image generator (such as a digital still 
20 camera having embedded image processing capabilities), commands to a server 

computing device from a cUent computing device, printers and other like peripheral 
devices that generate system commands (for example, re-rasterizing a particular digital 
image to suit a particular printing need), and other devices well known to those skilled in 
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the art. In the preferred embodiment, an open standard such as XML is desired. 
However, any other format, open or proprietary, can be used to represent the "edit hst". 

The image processing engine 212 operates on the digital image embedded in the 
incoming digital image stream 210 based, in part, upon the accompanying edit list data, if 
any. For example, in the case where the digital image embedded in the input stream 210 
is a digital negative type image (i.e., a reference image), then the incoming digital 
negative generally has no associated edit list data (since it is by definition an image to be 
used as a reference). This is typically the case when the image processing engine 212 
detects that the incoming digital image has no associated edit list data. In this case, the 
image processing engine 212 treats that incoming digital image as a reference digital 
image and passes it through to an output unit 216 without fiirther processing. 

When the image processing engine 212 detects that the incoming image is a 
resultant proxy image (i.e., one that has been modified), the image processing engine 212 
then determines the method of association (i.e., whether the corresponding edit list data is 
embedded in the incoming image file as an edit hst (as illustrated in Fig. 3 A), if the 
image file points to the appropriate edit list file (as illustrated in Fig. 3B), or has been 
generated by an external source (such as a UI control, printer, or other imaging device)) 
and not actually associated with the resultant image in any way. 

The edit list data associated with the resultant image is identified and known to be 
of the specific format known by the image processing engine 212. Of course, it is 
possible for a digital negative or the resultant image to include other data as required by 
an application that could be ignored by the image processing engine 212 if it is deemed 
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irrelevant. This may be the case when file formats are developed that embed other types 
of data in the edit list in its preferred embodiment as an XML file format. 

It is particularly advantageous that an edit Ust (commands for the specific imaging 
operations) of operations that are applied to a given photo are included in a particular edit 

5 list file can be stored with the resultant image. This arrangement allows for those 

operations identified by the edit hst to be reappHed to the digital negative, if desired. It 
also provides a way for the user to "undo" certain operations or "disabled" or otherwise 
turn off certain operations that have been appHed to the photo. In so doing, this 
mechanism allows these operations to be reappHed at any resolution such that higher 

1 0 quality results can be achieved. 

In some cases, however, the image-processing engine 212 detects external edit Hst 
data only that is supplied by, for example, a printer. In such cases, the image processing 
engine 212 is capable of using the external edit list data file to process any selected image 
file based upon the external edit Hst data. For example, if a particular printing job 

15 requires a printed image to be enlarged fi-om the original, then the associated edit Hst can 
have re-rasterizing instructions which have the effect of scaling the original image to 
whatever size is deemed to be appropriate. In the described embodiment, the image 
processing engine 212 processes the received digital image according to the instructions 
(if any) included in any edit Hst associated with (embedded in or pointed to by) the 

20 incoming image file. When the image processing engine 212 modifies an incoming 
digital image based upon acquired edit list data, the modified image can be either a 
modified resuHant proxy image with an embedded edit Hst or it can be a modified 
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resulting proxy image that points to an edit list file having the appropriate edit hst data in 
the form of an edit Hst, for example. 

In the case of a printer generated edit list and the incoming image file is a digital 
negative, then the image processing engine performs the image processing operations 
5 fi-om the edit hst (such as enlarging the image) on the incoming digital negative and then 
passes it to an output selector 214. In the cases where the incoming resultant image file 
has its own edit Ust either embedded in or pointed to, then the image processing engine 
212 performs the image processing operations accordingly on only those pixels in the 
incoming digital image so identified by the corresponding edit hst data in the edit list 
10 associated with the resultant image as well as those editing commands generated by the 
printer (such as enlarging the original). In both of these cases, a physical "resultant 
image" might not saved, but passed through to the printer port 220 for output. 

The use of the externally generated edit Hst could also serve other purposes. For 
example, an apphcation or device (such as a printer) could allow the user to toggle 
15 viewing/output display states. For instance, suppose every displayed output image 
should have an imaging operation appUed (such as color adjustment needed for a de- 
saturation step before display on an NTSC television). These commands could be use as 
input to the edit list source 213 and automatically applied to all photos processed. In 
some situations, where the images are not saved, but printed (or just displayed), a 
20 physical resultant image might not be persisted (beyond an intermediate image/file that is 
sent to the printer or display). 

In a similar way, the edit list source 213 could be driven by an apphcation or 
device to allow for batch operations. For example, when images are first uploaded to a 
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Web site, each photo could be automatically processed (made better via AutoFix imaging 
operation) when they are uploaded. However, in this case, the original digital negative 
could be preserved, but with a modified resultant image that contains an edit Ust of 
operations (such as AutoFix) with the link back to the original digital negative. 

5 In any case, all output image streams are output to the output selector 214 and 

based upon user supplied selection criteria, one of the output image streams is selected to 
form an output image stream 215. In the described embodiment, the output image stream 
215 forms an input to an output unit 216 that is coupled to external circuitry that can 
include an image display unit 218 capable of displaying the image provided by the output 

10 unit 216. In still other cases, the output unit 216 can be coupled to an output port 220 
capable of being coupled to other external circuitry (not shown) such as a computer that, 
in turn, provides an appropriate display apparatus. In the described embodiment, the 
output port 220 includes I/O ports such as a parallel port, a serial port, a system bus, a 
USB port, 1394, SCSI, a TV signal output port, a PCMCIA port, network port (to connect 

15 to the Internet), as well as a modem port. As such, the output port 220 can also be 

arranged to transmit the image provided by the output unit 216 (in a format such as raw 
RGB, JPEG, TIFF, PostScript, a proprietary image format, etc.) to an interconnected 
network of computers (such as the Internet). In this way, any of the interconnected 
computers can be used to, for example, view the image embedded in the image stream 

20 215 thereby allowing any number of end-users to view the embedded image. 

It is a particular advantage of the inventive digital image processor system 200 
that the image is processed (rendered) at whatever resolution is required by a particular 
output device (i.e. display, printer, etc.). The mechanism for accessing and rendering of 
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an image to the desired resolution is through an encoding mechanism that hnks the digital 
negative and an edit-list of operations that have been apphed to the resuhant proxy 
image. This provides the benefit for the user to view (and process) a much smaller proxy 
image file as necessary, but with the ability to access the original digital negative and 
rasterize it at any resolution. Furthermore, the proxy image (if it is stored in a JPEG 
format or an alternate format as desired) can serve both as a container that holds the 
reference to the digital negative and the edit hst as well as a standard image file that can 
be viewed using a conventional application. 

Fig. 2 illustrates a particular implementation of the image processing engine 212 
in accordance with an embodiment of the invention. As implemented, the image 
processing engine 212 includes an input controller 250 coupled to an image processor 
252 and an edit list processor 254. In operation, the input controller 250 determines 
whether or not a received input signal is a resuhant proxy image, a reference image, or an 
external edit list data file. In the case when the input signal is a reference image (digital 
negative) and no external edit list data is supplied, the input controller 250 outputs the 
reference directly to the output unit 216 without further image processing. 

In the case when the input signal is a reference image (digital negative) and there 
is external edit list data is suppUed (such as from a printer), the input controller 250 
passes the appropriate edit list data to the edit hst processor 254. When edit hst data is 
supplied, the edit Ust processor 254 will ask the input controller to request the necessary 
files (other referenced edit hsts). Once the input controller obtains the necessary files, 
they will be passed to the image processor 252. The image processor 252 then uses the 
image processing operations provided by the edit hst processor as defined by the 
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embedded edit list data to modify the original digital negative image to form a resulting 
proxy image with embedded edit list data in one case or in another case a resulting proxy 
image that points to an external edit list data file provided by the edit list processor 254. 
On the other hand, if the input signal is a resultant proxy image with embedded 
5 edit list data, then the input controller 250 passes the appropriate edit Hst data to the edit 
Hst processor 254. When edit list data is supplied, the edit hst processor 254 will 
determine from the edit list data how to access the original digital negative and ask the 
input controller to request the necessary files. Once the input controller obtains the 
necessary image files, they will be passed to the image processor 252. The image 
10 processor 252 then uses the image processing operations provided by the edit list 

processor as defined by the embedded edit list data to modify the original digital negative 
image to form a resulting proxy image with embedded edit hst data in one case or in 
another case a resulting proxy image that points to an external edit list data file provided 
by the edit list processor 254. 
15 When the edit list processor 254 must process several different edits Ust (such as 

one referenced by the resultant image (or possibly multiple ones recursively referenced) 
or fifom an external device (such as a printer), the edit Hst processor will have to generate 
an accumulated edit Ust that represents sum of all edit operations. In the preferred 
embodiment, XML is used, however, other format are equally vaHd. If there are 
20 conflicts between the various edit lists, the apphcation will need to reconcile these 
differences and process the edit hst as is required. 

In those cases where the resulting proxy image points to an edit hst file or where 
the edit list data is in the form of an external edit hst file, the image processing engine 
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212 uses the pointer to generate a reference (such as a URL) to locate and download the 
appropriate edit list file(s). The retrieved edit hst data is then used by the edit hst 
processor 252 to generate the resulting proxy image accordingly. 

It should be noted that if the input controller 250 cannot access the original digital 
5 negative(s), other muhi-media assets, or retrieve other parts as described in the edit list, 
the image processor 252 can, at the direction of the controlling application, directly 
output the original inputted proxy image. In this way, the user will still see a version of 
the output image, even though it has not been re-processed. 

It should also be noted that if the image processor 252 cannot process the 
10 commands specified by the edit hst processor 254 for what ever reason, the image 
processor 252 can, at the direction of the controlling application, directly output the 
original inputted proxy image. In this way, the user will still see a version of the output 
image, even though it has not been re-processed. 

It should also be noted that the edit Ust data associated with a reference proxy 
15 image can point to multiple digital negatives and other multimedia content used in the 
creation of the reference proxy image. This can include vector images (such as 
PostScript, EPS, or PDF), raster images (such as JPEG, TIFF, or PNG), text strings, 
video, soimd, and the like. 

Referring now to Fig. 4 showing a distributed system 400 in accordance with an 
20 embodiment of the invention. In the distributed system 400, the image processing engine 
212 is coupled to a host computer 402-1 by way of the output port 220. The host 
computer 402-1 can be, in turn, coupled to a server computer 404 by way of a network 
406 or other host computers 402-2 through 402-n or an imaging appliance 403-1 through 

DIGIP013/MJF 28 PATEN! 



403-n. It is also possible that an imaging appliance 403-1 (such as a digital still or digital 
video camera, set-top box, game machine, or photo appliance), which has embedded 
inside of it an image processing engine 212, can be coupled to a server computer 404 by 
way of a network 406 or other host computers 402-1 through 402-n or other imaging 

5 appUances 403-2 through 403 -n. 

Further, it is also possible that the image-processing engine 212 can be directly 
connected to the network 406, independent of computer 402-1 through 402-n. 

The network 406 can take the form of a distributed network of interconnected 
computers, such as can be found in the Internet, or it can be a small scale intranet type 

10 system found in, for example, small companies and the like. In the described 

embodiment, the server computer 404 can include a server image processor unit 408 
arranged to perform high speed image processing operations based upon edit list data, for 
example, found in an edit Ust associated with a particular image file. The server image 
processor unit 408 is arranged to perform those image processing operations that are 

15 either too time consuming or resource intensive to be efficiently performed on the host 
computer 402-1 or the imaging appliance 403-1. 

One such case is when a high resolution rendered image with all operations 
appHed (with maximum quahty imaging algorithms) is stored in the standard resultant 
image file (i.e. JPEG). Associated with this JPEG file (either embedded or linked 

20 externally) is the edit Ust (commands for the specific imaging operations) and reference 
to the original digital negative. Using this arrangement, a conventionally configured 
apphcation can ignore the edit list and digital negative reference and simply display the 
resuhant image. It is one of the advantages of this invention that allows an application or 
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other device that does not implement this described system to still have the ability to 
display the proxy resultant image (at whatever resolution it was saved). However, the 
appHcation or other device that does not implement this described system can not take 
advantage of all the other advantages of this invention. 

5 In the described embodiment, the digital image processing engine 2 1 2 can be 

included in any of a number of digital appliances 403-1 such as for example, a digital still 
or digital video camera, set-top box, game machine, photo apphance, and the like. Using 
the example of a digital still camera, instead of storing the high-resolution resuhant 
image, an application may decide to store only the low-resolution resultant image. For 

10 the case of a digital camera based application, this may be more desirable given the 
memory and processor constraints of the embedded system. For this case, the user can 
still choose the image operations as specified in the edit list, but only a low-resolution 
resuhant image is generated and stored locally, in addition to the original digital negative 
In some cases, such as when the digital image processing engine 212 is part of a 

15 set-top box with limited or no local storage space, all components, both digital negative 
and the low-resolution resultant image with its edit list could be stored remotely on the 
network. 

This has several benefits. First, the amount of processing power (and internal 
memory requirements) of either the local computing device or the remote computing 
20 device (where ever the image processing occurs) is reduced since much fewer pixels are 
processed. Second, the size of the resuhant image file is much smaller since only a low- 
resolution (or thumbnail) version of the image is persisted. Most importantly, the 
original digital negative is preserved, along with the smaller resultant image. This is 
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critical since digital camera storage is very constrained. In this case, the original digital 
negative will be saved and a lower-resolution (and much smaller) resultant image with its 
associated edit list will be saved to the digital still camera. 

Once the image has been processed, the low-resolution resultant image can be 

5 sent over the network 406 to any number of other host computers 402- 1 through 402-n or 
imaging appliances 403-1 though 403-n for viewing on their respective displays. This has 
the advantage of only consuming a limited amount of bandwidth for the transfer of the 
lower-resolution thumbnail image through a low-bandwidth connection. However, at the 
request of the application, the original digital negative could be accessed such that the 

10 edit operations could be reapplied to generate a higher-resolution resultant proxy image. 

Depending on the capabiHties of the host computer or the imaging appliance, this 
processing could either occur on the host computer 402-1 or imaging appliance 403-1, or 
on one of the system attached to the host computer via the network 406, such as another 
the server image processor unit 408 that is part of the server 404. 

1 5 It should be noted that for either case described above, the file that contains the 

edit-hst could take on several forms. It may be desirable to generate a JPEG file that 
contains the resultant image and embeds the edit-hst and the reference to the digital 
negative. This has the benefit that the user has one encapsulated file to maintain. 
Another approach involves creation of an edit list file that contains the edit-list, the 

20 reference to the digital negative, and a reference to the resultant image (possibly a JPEG 
file that may or may not include metadata about the edit-hst and/or reference to digital 
negative). A third approach would involve the complete encapsulation of not only the 
resultant image and the edit list, but also a digital negative. This could take the form of 
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an image file (JPEG, TIFF, etc.) or a general "collection" file (such as a compressed ZIP 
or JAR (Java Archive)). This may be desirable as a distribution mechanism, but 
maintains the properties described by this invention. It should also be noted that with any 
of these variants, multiple resolutions may also be stored (embedded or linked) at the 

5 application's discretion. 

With conventional Web editing techniques, if a user is editing a photo that resides 
on the Web, the entire image must be downloaded and processed locally. However, this 
invention allows for a more efficient alternative approach that calls for a low resolution- 
image to be downloaded from the remote computing device (such as the server computer 

10 406) and processed by the local computer device (such as the host computer 402-1 or 
photo apphance 403-1). The higher resolution image data would only be required by 
local computing device and downloaded from the remote computing device if needed. 
For example, referring to Fig. 4, a user working on a local computing device (such as a 
photo appliance 403-1 without access to local storage) can efficiently save changes to the 

1 5 server computer 404 over the network 406 even when the network 406 is a low 

bandwidth network such as a modem. This results in much less bandwidth, assuming 
only a low-resolution resultant image is transmitted with the edit Ust and other metadata 
to the remote computing device (such as the server computer 406). Alternatively, the 
user may choose to save the resuhant image locally with the edit list, in for example, the 

20 host computer 402-1 with a link to the original digital negative that can be reference at 
any node in the network 406. 

Another advantage of the distributed nature of this invention is that the actual 
image operations can be processed either locally in any of the local computing devices 
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(such as the host computer 402-1 or photo appHance 403-1) or a remote computing 
device (such as the server compute 404), as required by the apphcation. It also is 
possible to accommodate a mixed mode model in which the low-resolution image is 
processed locally (performing the commands in the edit list), but a high-resolution image 
is processed using the same edit Hst remotely on the server computer 404. 

The invention will now be described in terms of a digital appliance, such as a 
digital camera having relatively small LCD screens for display and viewing. Since, the 
LCD screens are very small and only require a low-resolution image, processing of the 
image is only performed on a thumbnail or low-resolution image and the digital negative 
will be saved to the camera's memory card. 

Referring to Fig. 5, a digital camera system 500 in accordance with an 
embodiment of the invention is described. In this example, the camera system 500 is 
capable of color correcting digital images either on the fly or after the fact. In some 
cases, it may be advantageous to generate an edit list describing the color correction, such 
as an ICC profile, that is required in conjunction with an uncorrected reference image, 
thus preserving the original digital negative. Also stored will be a resultant image, most 
likely of a low-resolution thumbnail image with the color correction applied such that it 
can be displayed on the LCD of the camera. The uncorrected reference image (digital 
negative) along with the edit list can then be downloaded to either a local host computing 
device or can be sent to a remote computing device for further image processing. 
Alternatively, if transmission of the digital negative is not possible, the resultant proxy 
image, along with the edit list, can be transmitted and displayed on local host computing 
device or remote computing device. At a later time, the edit list, which is associated with 
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the resultant proxy image can be processed and applied to the digital negative, which is 
referenced from the edit list. 

Typically the digital camera system 500 includes the image source 202 that 
includes an optical lens 502 for focusing light rays onto a imaging capture unit 504. The 
5 imaging capture unit 504 typically rehes upon an array of light sensitive photo optic cells 
506 capable of converting the light rays, in the form of photons, received from the optical 
lens 502 into representative analog signals. Most commonly, the photo optic cells take 
the form of charge coupled devices (CCDs), although other devices such as CMOS 
receptors may be used as well. 
10 As is well understood in the art, each CCD array 506 has associated with it a 

specific color filter array (CFA) 508. In most applications, the CFA 508 is an empirically 
derived pattern of individual color filters each associated with a specific CCD cell in the 
CCD array. When the analog signals representative of the captured image are produced 
by the image capture unit 504, they are sent first to an analog to the digital (A/D) 
15 converter unit 208 which converts the representative analog signals into digital signals 
representative of the image. The digital signals are then passed to a digital signal 
processor (DSP) 518 where they are converted to an appropriate digital format. 

In some embodiments of the invention, an image compression circuit 520 as well 
as a memory 522 both receive the appropriately formatted digital signals. The image 
20 compression circuit 520 operates to digitally compress the received digital images in 
order to reduce the amount of resources required to further process the digital signals. 
One such formatting scheme referred to as JPEG is commonly used, although there are a 
wide variety of suitable picture formats. 
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In one embodiment, the digital image stored in the memory 522 can be sent by 
way of a memory interface (not shown) to a memory slot 524 capable of receiving a 
memory card suitable for storing the processed digital signals. Such memory cards 
include "floppy" disks, flash EPROM cards, R/W compact disc (CD), SmartMedia, 
CompactFlash, Sony's MemoryStick, DVD, DAT, and the like. Alternatively, the digital 
image stored in the memory 522 can be sent to external circuitry, such as a computer, by 
way of the output port 220 for viewing or, in some cases, additional image processing. In 
the situation where a number of images are stored in the memory 522, a user can pre- 
select any number of images for additional processing by viewing the images (generally 
in a thumbnail form) using a viewing device such as a viewfinder 526. 

In the case where the digital image is to be processed by an external computer, for 
example, an edit list generator 523 generates an edit list consistent with those color 
corrections required to compensate for the ambient light conditions. The edit list 
generator 523 can either embed the edit list in the reference image or it can create a 
separate edit list which is associated with the reference digital image, by for example, a 
pointer. 

If it is determined that the digital image is not to be processed by the image 
processing imit 212, then the image stored in the memory 522 can be directly sent to 
either the display unit 526, to a device coupled to the memory slot 524, or to external 
circuitry, such as a computer, by way of the output port 220. In this case, the external 
computer can actually perform the color correction as indicated by the edit Ust generated 
by the edit list generator 523. 
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In the case where the output device includes a printer, the user is able to print out 
the photos captured with the digital camera 500. If the user is printing out a digital 
negative, those photos can be directly printed as specified by the user. However, if the 
user prints out photos that have been modified that contain only a low-resolution 

5 thumbnail, it may not be at a high enough resolution to meet the needs of the printer. For 
this scenario, the printer can re-rasterize the image by applying the edit list to the original 
digital negative to obtain high quality output results as desired. 

It should be noted that the invention also provides for a very efficient means of 
describing a print job that allows rasterization to occur at the printer. Typically, an 

10 application (or print driver) rasterizes an image (or composition of images) to the specific 
print resolution locally on the client, that is then output to the printer. While this 
approach is acceptable for local, cUent-side printing, it is clearly inefficient for server- 
side remote printing. 

An alternative approach involves sending the edit list and references to the digital 

15 negative(s) (the contents associated with the resultant image), to the server for 

processing. These operations can then be reapplied to the digital negative to produce the 
desired print resolution on the server. This approach clearly preserves bandwidth 
because the fully-rasterized image is not sent to the print server. This is particularly 
important if the print job involves any vector-based artwork that contains a highly 

20 compressed description of a page layout. 

Additionally, the invention is well suited for printing of "package prints." As 
well known in the art, package prints are those in which one photo is printed on a variety 
of different print layout templates (i.e. one 5x7 and two 3x4). If only one photo is 
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printed, but to several different orientations and sizes, only one copy of the photo must be 
transferred. Equally important, it reduces the amount of processing power required on 
the chent side since the rasterization can be offloaded to the server. 

Referring now to Fig. 6, a flowchart is shown detailing a process 600 processing a 

5 digital image by the digital image-processing engine in accordance with an embodiment 
of the invention. The process 600 begins at 602 when a particular digital image is opened 
by the image-processing engine. At 603, a determination is made if the image should be 
directly displayed, without any further processing, in which case control is passed to 630 
where the image is passed to the output device for display (or print or to a file). This 

10 could occur in several circumstances. If the application does not have knowledge of this 
invention, it will simply display the image. If the application does have knowledge of 
this invention, but wants to quickly display the contents of the image, whether it is a 
resultant proxy image or the digital negative (an image without any associated edit list), it 
can b5q3ass any additional processing and simply display the image. 

15 If the image should not be directly displayed, then at 604 and 606, a determination 

is made if there is an edit list data embedded in or if external edit hst data is referenced 
to, respectively, the opened image file. If it is determined that there is no embedded or 
referenced edit hst data with the associated digital image, then control is passed to 630 
where the image is passed to the output device for display (or print or to a file). 

20 If it was determined that there was either embedded or referenced edit list data 

associated with the input digital image, then at 610 a determination is made to what the 
desired output resolution should be. Once this is known, then at 612 the first edit list 
element is processed and any necessary digital negatives are retrieved at the resolution 
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appropriate based on the value determined in 610. At 614 a determination is made to see 
if additional edit list elements or more digital negative elements are to be retried, and if 
so, control is passed back to 612. Otherwise, control is pass to 616 where the edit list 
elements and digital negative(s) are processed, based on the desired resolution 
5 determined in 61 0, resulting in a new output image. 

At 61 8, a determination is made whether or not the edit hst data associated with 
the resultant output image generated at 616 should also be generated. In many cases, this 
will simply be the original edit list processed at 612. If no edit list data is to be 
generated, the image generated from 616 is passed to 630 where the image is passed to 
10 the output device for display (or print or to a file). 

If an edit list is to be generated, a determination is made at 620 whether it should 
be embedded with the image file or external as a separate file. If the edit Ust data is 
determined to be embedded, then the edit Hst data is embedded in the resultant proxy 
image at 622, otherwise the edit list data is stored in an associated edit hst file, such as, 

15 for example, an edit hst at 624. Regardless, control is then pass to 630 where the image 
is passed to the output device for display (or print or to a file). 

It should be noted, if the resolution generated (and requested at 610) does not 
exactly match the output device, it may be necessary to resample the output image to 
match the output device. While the most acciirate results would be achieved if the image 

20 were re-rasterized, based on the edit hst and digital negative(s), this may not always be 
done. For example, when the user zooms in/out to/from an image, it may be desirable to 
quickly resample the image to get the desired resolution as needed. 
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In a particular scenario, a user creates an intelligent "e-card" as detailed by 
flowchart of a process 700 as illustrated in Fig. 7 in accordance with an embodiment of 
the invention. The process 700 can be carried out in any suitably arranged digital 
appliance, such as for example, the digital imaging set-top device 500 described above 

5 with reference to Fig. 4 in conjunction with the distributed network 400. Therefore, a 
user chooses a postcard, for example, at 702 while choosing photo image(s) at 704 to be 
included in the postcard. Once selected, the user positions the photo images inside the 
selected postcard at 706, which may involve zooming and panning the photo inside the 
composition. Once satisfied, the composition is saved as a single image at 708 at a low 

10 resolution. In addition at 708, the edit hst will also be saved, either embedded inside or 
associated with, the generated resultant proxy image. Once saved, the user can email the 
image to the receiver(s) at 710. Control is then passed to 602 when the image can be 
opened 

Fig. 8 illustrates a process 800 for facilitating efficient video editing in a 
15 distributed environment in accordance with an embodiment of the invention. At 802, a 
user drops off a videotape (in digital or analog format) to a video processor or merchant 
that supports this process. At 804, a determination is made if the videotape is in analog 
format or digital format. If in analog format, the tape is converted into digital format at 
806. In either case, control is then pass to 808 where a low-resolution digital video 
20 stream is created as well as determining at 810 whether a higher resolution video stream 
is to be created If a higher resolution video stream is to be created, then the higher 
resolution video stream is created at 812, otherwise, the low resolution video stream is 
either placed on a CD, DVD, or other digital media (or uploaded to the network where it 
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can be accessed) at 814. At 816, the user accesses the low-resolution digital video stream 
and performs the necessary edits (most likely from an video appHcation or consumer 
electronics device at home or a kiosk in a store). 

At 818, these edits are stored an edit list in an appropriate format (such as XML or other 
5 compact open format) such that they can be provided to a video processor. At 820, the 
video processor accesses the edit list generated in 818, either through the network or 
some read-write media provided by the customer. At 822, a determination is made 
whether or not a higher resolution video stream is available (i.e., made available at 812) 
or is required at a different resolution than is currently available. If a higher resolution 
10 video stream is not currently available or a different resolution is required, then the 
higher resolution or different resolution video stream is created at 824. In any case, at 
826, the higher resolution video stream is retrieved and at 828, the edit list is applied to a 
higher-resolution video stream (i.e., rasterization of video stream based upon the edit list 
at the appropriate resolution). At 830, the merchant outputs the results from the 
15 rasterization of the edit list at the desired resolution to an appropriate output medium, 
such as DVD, CD, digital or analog tape, so the user can view the edits they had 
performed on the low-resolution video stream, but rendered at a much higher resolution 
output. 

Fig. 9 illustrates a computer system 900 or any imaging-enabled computing 
20 appliance /device that employed to implement the invention. The computer system 900 
or, more specifically, CPUs 902, may be arranged to support a virtual machine, as will be 
appreciated by those skilled in the art. As is well known in the art, ROM acts to transfer 
data and instructions uni-directionally to the CPUs 902, while RAM is used typically to 
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transfer data and instructions in a bi-directional manner. CPUs 902 may generally 
include any number of processors. Both primary storage devices 904, 906 may include 
any suitable computer-readable media. A secondary storage medium 908, which is 
typically a mass memory device, is also coupled bi-directionally to CPUs 902 and 

5 provides additional data storage capacity. The mass memory device 908 is a computer- 
readable medium that may be used to store programs including computer code, data, and 
the like. Typically, mass memory device 908 is a storage medium such as a hard disk or 
a tape which generally slower than primary storage devices 904, 906. Mass memory 
storage device 908 may take the form of a magnetic or paper tape reader or some other 

10 well-known device. It will be appreciated that the information retained within the mass 
memory device 908, may, in appropriate cases, be incorporated in standard fashion as 
part of RAM 906 as virtual memory. A specific primary storage device 904 such as a 
CD-ROM may also pass data uni-directionally to the CPUs 902. 

CPUs 902 are also coupled to one or more input/output devices 910 that may 

15 include, but are not limited to, devices such as video monitors, track balls, mice, 

scanners, PC- card readers, digital cameras, digital video, remote control units for TVs, 
VCRs, DVD players, and the like, keyboards, microphones, touch- sensitive displays, 
transducer card readers, magnetic or paper tape readers, tablets, styluses, voice or 
handwriting recognizers, or other well-known input devices such as, of course, other 

20 computers. Finally, CPUs 902 optionally may be coupled to a computer or 

telecommunications network, e.g., an Internet network or an Intranet network, using a 
network connection as shown generally at 912. With such a network connection, it is 
contemplated that the CPUs 902 might receive information firom the network, or might 
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output information to the network in the course of performing the above-described 
method steps. Such inforination, which is often represented as a sequence of instructions 
to be executed using CPUs 902, may be received from and outputted to the network, for 
example, in the form of a computer data signal embodied in a carrier wave. The above- 
5 described devices and materials will be familiar to those of skill in the computer 
hardware and software arts. 

While the present invention has been described as being used with a digital image 
or video system, it should be appreciated that the present invention may generally be 
implemented on any suitable digital image system. Additionally, this invention is not 
10 hmited to digital images, but extends to any image type where a high-resolution data 

object representation can be approximated by a lower-resolution proxy image, including 
video imaging application and systems as well as audio systems. 

It should be noted that the digital negative itself could be a standard image file 
type (JPEG, TIFF, PNG, etc.) or a propriety image format. Alternatively, the digital 

15 negative could be a recursive resultant image that points to multiple digital negative(s) 
and its associated edit list. This provides a powerful mechanism in which a low- 
resolution proxy image could be subsequently edited and/or placed inside a print job, 
package print, card, or digital album. Then an application, or server application, that 
supports this invention will be able to reapply all the editing operations recursively to 

20 generate the highest quality output image as required by the output device. For those 
applications that do not support this invention, they will be able to print the top most 
resultant image that is the composition of all recursively added inner images. 
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The resultant proxy image may also be a standard image file type (JPEG, TIFF, 
PNG, etc.) or a proprietary image format, with a resolution specified by the application 
that creates the resultant proxy image. It is also possible in some cases that a resuhant 
proxy image does not even exist (or is "empty" with out a physical pixel representation). 
5 As described above, the edit list of operations may describe a print layout template with 
the associated digital negatives (and possible edit list of editing operations) and other 
multimedia assets. In this case, the creation of the resultant image is not needed since the 
printer service will process the edit list and its associated digital negatives and other 
assets and generate the desired output. As a convenience, this could be represented as a 
10 blank JPEG file with an embedded or externally linked edit list. 

It should be also noted that the edit list could be persisted through many different 
mechanisms. The preferred embodiment uses XML (extensible Markup Language), but 
other formats, both standard and proprietary could be used to implement this invention. 
Further, this edit list can either be included with the resultant image or linked to it via a 
15 separate file. 

The distributed nature as described in the recursive example above also provides a 
powerful and flexible workflow such that separate designers can work on different parts 
of this recursive structure without effects to the others. 

Therefore, the present examples are to be considered as illustrative and not 
20 restrictive, and the invention is not to be limited to the details given herein, but may be 
modified within the scope of the appended claims along with their full scope of 
equivalents. 

What is claimed is: 
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In the claims: 

1 . In a distributed computing system, a method of sharing a multi-media 
object between a first node and a second node, comprising: 
at the first node, 

5 storing a digital negative of the multi-media object, wherein the digital 

negative represents the multi-media object at a first resolution and a first format; 

modifying the digital negative to form a resultant image at a second 
resolution and a second format ; 

associating an edit list based on the modifying with the resultant image; 
10 linking the edit list to the digital negative; 

at the second node, 

fetching the resultant image; 

determining an output resolution and an output format of the resultant 

image; 

15 converting the resultant image to the determined output resolution and the 

determined output format; and 

outputting the resultant image at the determined output resolution and 

output format. 

20 2. A method as recited in claim 1 wherein the first resolution is a higher 

resolution than the second resolution. 
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3. A method as recited in claim 2, wherein the converting comprises: 

if the determined output resolution is the second resolution, then 
outputting the resultant image at the second resolution. 

4. A method as recited in claim 3, wherein the converting further comprises: 

5 if the determined resolution is a third resolution that is different than the second 

resolution, then 

fetching the edit list; 

fetching the digital negative linked to the edit hst; 

operating on the digital negative to form the resultant image at the third 
10 resolution based upon the edit list. 

5. A method as recited in claim 3, wherein the first node is a first computing 
device coupled to a first input device and a first output device and wherein the second 
node is a second computing device coupled to a second output device and a second input 
device. 

15 6. A method as recited in claim 5, wherein the first computing device and the 

second computing device are linked in a peer-to-peer arrangement. 

7. A method as recited in claim 5, wherein the first computing device and the 
second computing device are wirelessly linked. 

8. A method as recited in claim 7, wherein the converting is performed at the 
20 second computing device. 
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9. A method as recited in claim 8, wherein the multi-media object is a digital 
image formed of a plurality of pixels. 

10. A method as recited in claim 9, wherein the converting is performed on a 
subset of the plurality of pixels based upon the edit list and the third resolution thereby 
preserving transmission resources required to link the first and the second computing 
devices. 

11. A method as recited in claim 1, wherein the first format is selected from a 
group comprising: JPEG, TIFF, and PNG. 

12. A method as recited in claim 1, wherein the second format is selected from 
a group comprising: JPEG, TIFF, and PNG. 

13. An apparatus for sharing a multi-media object between a first node and a 
second node, comprising: 

at the first node, 

a means for storing a digital negative of the multi-media object, wherein 
the digital negative represents the multi-media object at a first resolution and a first 
format; 

a means for modifying the digital negative to form a resuhant image at a 
second resolution and a second format; 

a means for associating an edit list based on the modifying with the 
resultant image; 

a means for linking the edit list to the digital negative; 
at the second node, 

a means for fetching the resultant image; 
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a means for determining an output resolution and an output format of the 
resultant image; 

a means for converting the resultant image to the determined output 
resolution and output format; and 

a means for outputting the resultant image at the determined output 
resolution and output format. 

14. An apparatus as recited in claim 13 wherein the first resolution is a higher 
resolution than the second resolution. 

15. An apparatus as recited in claim 14, wherein the means for converting 
comprises: 

a means for outputting the resultant image at the second resolution if the 
determined output resolution is the second resolution. 

1 6. An apparatus as recited in claim 1 5, wherein the means for converting 
fiirther comprises: 

a means for fetching the edit hst; 

a means for fetching the digital negative linked to the edit list; and 

a means for operating on the digital negative to form the resultant image at 

the third resolution based upon the edit list if the determined resolution is a third 

resolution that is different than the second resolution. 
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17. An apparatus as recited in claim 14, wherein the first node is a first 
computing device coupled to a first input device and a first output device and wherein the 
second node is a second computing device coupled to a second output device and a 
second input device. 

18. An apparatus as recited in claim 1 7, wherein the first computing device 
and the second computing device are linked in a peer-to-peer arrangement. 

19. An apparatus as recited in claim 1 7, wherein the first computing device 
and the second computing device are wirelessly linked. 

20. An apparatus as recited in claim 1 9, wherein the means for converting is 
coupled to the second computing device. 

21. An apparatus as recited in claim 20, wherein the multi-media object is a 
digital image formed of a plurality of pixels. 

22. An apparatus as recited in claim 21, wherein the means for converting is 
coupled to a subset of the plurality of pixels based upon the edit fist and the third 
resolution thereby preserving transmission resources required to link the first and the 
second computing devices. 

23. An apparatus as recited in claim 13, wherein the first format is selected 
from a group comprising: JPEG, TIFF, and PNG. 

24. An apparatus as recited in claim 1 3, wherein the second format is selected 
from a group comprising: JPEG, TIFF, and PNG. 
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Techniques for generating a distributed low- 
resolution DIGITAL IMAGE INA COMMON FORMAT CAPABLE OF 

viewing in any resolution 



ABSTRACT OF THE DISCLOSURE 

In one embodiment, in a peer-to-peer connected environment, a method of sharing 
a multi-media object between a first node and a second node is disclosed. At the first 
node, a digital negative of the multi-media object is stored, wherein the digital negative 
represents the multi-media object at a first resolution. The digital negative is modified to 
10 form a resultant image at a second resolution, an edit list based on the modifying is 

associated with the resultant image, the edit Hst is linked to the digital negative. At the 
second node, the resultant image is fetched at the second resolution, an output resolution 
of the resultant image is determined, the resultant image is converted to the determined 
output resolution, and the resultant image is output at the determined output resolution. 
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Fig. 7 
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